# Replication package for "Face detection, tracking, and classification from large-scale news archives for analysis of key political figures"

---
### Contributors:
  - Andreu Girbau
  - Tetsuro Kobayashi
  - Yusuke Matsui
  - Benjamin Renoust
  - Shin'ichi Satoh
---

## Overview

The code in this replication package for "Face detection, tracking, and classification from large-scale news archives for analysis of key political figures" using Python.
Experiments Tables 3-7 and Figures 5-7 from the paper, and Tables 1-3 and Figures 1-3 from the supplementary can be reproduced by running "run.sh" inside /code. Running all the replication code takes around 2 hours and 20 minutes.

After the successful run, the results for each table of the paper will be in **"/results/tabN"**, where N is a number between 3-7 depending on the corresponding table in the paper. Similarly, the raw form of the figures will be in **"/results/figN"**, where the naming convention is **figN-channel-information**. Supplementary material tables and figures are preceded by "SM", e.g. SM_tab3.

*Example 1: the results in "results/tab3" correspond to the results of Table 3 in the paper.*
*Example 2: the results in "results/fig5-f1_CNN" corresponds to the F1 metrics curve for CNN of Figure 5 in the paper.*

Each value of Tables 3-7 is extracted from the configuration triplet **detector-feats-classifier**.

*Example: in results/tab4, NHK-YOLO-vote indicates that the result corresponds to ***NHK***, with ***YOLO*** detector and ***vote*** classification strategy.*

The whole script takes around 1:30 hours for the paper experiments (from downloading the dataset, tracking+classification, metrics, and generating the plots for different face sizes per channel), and 50 minutes for the supplementary material experiments.

## Data Availability and Provenance Statements

This paper contains analysis on copyrighted data of TV News, therefore raw video frames are not provided for the replication package. 
Instead, we provide pickle (.pkl) files containing the information of all detected faces in the video excerpts used to test our system. 

This information is the **faces bounding box** located in time and space, and the **face embeddings**, as described in the paper.

The experiments presented in the paper can be reproduced by using this intermediate representation of the data. 

### Statement about Rights

- [x] I certify that the author(s) of the manuscript have legitimate access to and permission to use the data used in this manuscript.
- [x] I certify that the author(s) of the manuscript have documented permission to redistribute/publish the data contained within this replication package.


### License for Data

The data are licensed under a MIT license. See LICENSE.txt for details.


### Summary of Availability

- [ ] All data **are** publicly available.
- [x] Some data **cannot be made** publicly available.
- [ ] **No data can be made** publicly available.

### Details on each Data Source

| Data.Name          | Data.Files  | Location                                        | Provided | Citation |
|--------------------|-------------|-------------------------------------------------|----------|-----|
| “Video frames”     | .jpg / .png | data/dataset/train/channel/year                 | FALSE    | This work |
| “Frames metadata”  | .metaPA22   | data/dataset/train/channel/year                 | TRUE     | This work    |
| “Annotations”      | .csv        | data/dataset/train/channel/year                 | TRUE     | This work    |
| “Face information” | .pkl        | data/results/channel/train/detector-features    | TRUE     | This work    |
| “Face models JP”   | .pkl        | data/resources/face_models/faces_politicians    | TRUE     | This work    |
| “Face models US”   | .pkl        | data/resources/face_models/faces_us_individuals | TRUE     | This work    |

### Dataset collected by the authors

The annotations for the Japanese TV data (NHK, Hodo Station) used to support the findings of this study is publicly available.

### Dataset provided by Hong et al. (2021)

The annotations for the US data (CNN, FOX, MSNBC) used to support the findings of this study is property of Hong et al. (2021), therefore not provided in the public reproduction package of this work.

## Software requirements

Our code is tested with python3.9 in a CodeOcean environment. The libraries required are specified in the docker file configuration.

## Description of programs/code

Code is included in the `src` folder.

- Code in `face_classifier.py` tracks and classifies the face detections along a video. 
- Code in `metrics.py` runs the evaluation for the specified options (detector + classifier). 
- Code in `convert_results_to_fiftyone.py` converts the datasets to "fiftyone" format in order to reproduce figures 5-6.  
- Code in `convert_dataset_to_fiftyone.py` populates the dataset with the detections/classification of the key individuals for reproducing figures 5-6.
- Code in `run_fiftyone_f1_map.py` reproduces figures 5-6 in their raw form, generating the individual and per-channel F1 and mAP plots for different face sizes.
- Code in `analysis.py` reproduces Table 7, and Figure 7a/7b in its raw form, generating the per-channel, 15-day or 90-day average plots for normalized screen time, and NHK vs. HODO Station ratio.
- Code in `parse_results_to_tables.py` parses the results in "/results/output" to a similar structure that matches the tables of the paper.
- The rest of files contain classes and functions used by the previously mentioned scripts.

### License for Code

The code is licensed under a MIT license. See [LICENSE.txt](LICENSE.txt) for details.

## Instructions to Replicators

By running "run.sh" the results of the paper can be replicated. Inside "/code/run.sh", there are detailed comments that explain what each section of the script does to reproduce our results.

*Note: Table 7 and Figure 7 correspond to the screen time of key figures over a test set. This is, more than 10 years of daily videos per channel. As the data is 90+GB, and the computation takes 2-3 days, we provide our pre-computed results to generate Table 7 and Figure 7. In case of wanting to compute all tracking and classification within this capsule, set "flag_all=true" inside /code/run.sh, but to do so, you will need special permission from CodeOcean, as, by default, offers 20GB of storage space* [Link](https://help.codeocean.com/en/articles/1053605-quota-measurement-usage-variation-between-runs-and-what-if-i-run-out)

## List of tables and programs

| Results file name  |   Paper  |    Description                                  |
|--------------------|----------|-------------------------------------------------|
| tab3               | Table 3  | Missed detections percentage per TV channel.    |
| tab4               | Table 4  | Evaluation of our method over the three US TV cable news channels.    |
| tab5               | Table 5  | Comparison of our method between two Japanese TV programs, NHK News 7 and Hodo Station.    |
| tab6               | Table 6  | erformance with or without face tracking for different detectors.    |
| tab7               | Table 7  | Screen time of major US candidates.    |
|--------------------|----------|-------------------------------------------------|
| SM_tab1               | Supplementary Table 1  | Performance with different voting thresholds of US TV data.    |
| SM_tab2               | Supplementary Table 2  | Performance with different voting thresholds for Japanese TV data.   |
| SM_tab3               | Supplementary Table 3  | Performance of our system using the same FaceNet features as Hong et al. (2021).   |
|--------------------|----------|-------------------------------------------------|
| fig5-f1_CNN               | Figure 5  | F-score curve for CNN.    |
| fig5-map_CNN               | Figure 5  | mAP curve for CNN.    |
| fig5-f1_FOX               | Figure 5  | F-score curve for FOX.    |
| fig5-map_FOX               | Figure 5  | mAP curve for FOX.    |
| fig5-f1_MSNBC               | Figure 5  | F-score curve for MSNBC.    |
| fig5-map_MSNBC               | Figure 5  | mAP curve for MSNBC.    |
|--------------------|----------|-------------------------------------------------|
| fig6-f1_NHK               | Figure 6  | F-score curve for NHK.    |
| fig6-map_NHK               | Figure 6  | mAP curve for NHK.    |
| fig6-f1_HODO               | Figure 6  | F-score curve for HODO.    |
| fig6-map_HODO               | Figure 6  | mAP curve for HODO.    |
|--------------------|----------|-------------------------------------------------|
| fig7a-NHK-15-gov_opp-ratio  | Figure 7a  | 15-day average screen time for NHK of incumbent party leaders among all leaders.    |
| fig7a-NHK-90-gov_opp-ratio  | Figure 7a  | 90-day average screen time for NHK of incumbent party leaders among all leaders.    |
| fig7a-HODO-15-gov_opp-ratio  | Figure 7a  | 15-day average screen time for HODO of incumbent party leaders among all leaders.    |
| fig7a-HODO-90-gov_opp-ratio  | Figure 7a  | 90-day average screen time for HODO of incumbent party leaders among all leaders.    |
| fig7b-NHKvsHODO-15-gov_opp-ratio  | Figure 7b  | 15-day average screen time for NHK/HODO ratio of the share of incumbent party leaders.    |
| fig7b-NHKvsHODO-90-gov_opp-ratio  | Figure 7b  | 90-day average screen time for NHK/HODO ratio of the share of incumbent party leaders.    |
|--------------------|----------|-------------------------------------------------|
| SM_fig1  | Supplementary Figure 1  | Robustness check of Precision scores across voting thresholds.    |
| SM_fig2  | Supplementary Figure 2  | Robustness check of Recall scores across voting thresholds.    |
| SM_fig3  | Supplementary Figure 3  | Robustness check of F1 scores across voting thresholds.    |


The provided code reproduces:

- [X] All numbers provided in text in the Analysis section of the paper.
- [X] All tables and figures in the Analysis section of the paper.

## References

Hong, J., W. Crichton, H. Zhang, D. Y. Fu, J. Ritchie, J. Barenholtz, B. Hannel, et al. 2021. “Analysis of Faces
in a Decade of US Cable TV News.” In Proceedings of the 27th ACM SIGKDD International Conference on
Knowledge Discovery Data Mining. Association for Computing Machinery.
